RADON TRANSFORM ON REAL, COMPLEX AND 
QUATERNIONIC GRASSMANNIANS 



GENKAI ZHANG 

Abstract. Let G n ,k (K) be the Grassmannian manifold of fc-dimensional K-subspaces 
in K n where K = R, C, H is the field of real, complex or quaternionic numbers. For 
l<k<k'<n~ lwe define the Radon transform (lZf)(ri), r/ £ G n ,k'0&-), for 
functions /(£) on G ni fe(K) as an integration over all £ C rj. When k + k' < n we give 
an inversion formula in terms of the Garding-Gindikin fractional integration and the 
Cayley type differential operator on the symmetric cone of positive k x k matrices 
over K. This generalizes the recent results of Grinberg-Rubin for real Grassmannians. 



1. Introduction 

The Radon transform on rank one symmetric spaces has been studied extensively and 
is related to many areas in analysis and geometry; see |7j, jS] for a systematic treatment 
and e.g. [I], [TU], [T3|, [12], [Hj and references therein for some recent development. 
For higher rank symmetric spaces the theory is far from been complete and there has 
been comparably less progress. In a remarkable paper [3] Grinberg and Rubin find an 
inversion formula for the Radon transform from functions on the real Grassmannian 
G ni fc(M) of fc-dimensional subspaces in M n to functions on G n> fc'(R.), with 1 < k < 
k' < n — 1, k + k' < n, by using the Garding-Gindikin fractional integration on the 
space of real symmetric k x fc-matrices. It is natural to ask if the corresponding results 
hold for the Grassmannian manifolds over the complex and quaternionic numbers, 
and for corresponding non-compact symmetric spaces of matrix balls. In the present 
paper we will answer the question and prove the results for complex and quaternionic 
Grassmannians. We proceed with a brief summary of our results and the technical 
tools needed to prove them. 

Let K = R, C, H be the field of real, complex or quaternionic numbers with real 
dimension d = 1,2,4 and with the standard involution x — > x. Let G nj fc(lK) be the 
Grassmannian manifold of fc-dimensional subspaces over IK in K n . For 1 < fc < k! < 
n - 1 the Radon transform K : C°°{G ntk {K)) C°°(G n>k '(K)) is defined by 



;i.l) <f>(r]) = (Kf)(rj) = [ /(O^e, ieG n ,, 



where d v ^ is a certain probability measure on the set {£ G G nj fc(lK) : £ C 77} invariant 
with respect to the group of unitary transformations of 77; it can be defined using a 
group-theoretic formulation, see ()4.2j) . 
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When k = 1 the Grassmannian G nj fc(K) is the projective space of all lines £ in 
K n . The Radon transform /(£) — > 0(?y) is inverted by using the integration of (f)(7)) 
over all subspaces 77 that are at a fixed angle 9 with the given £, and by a fractional 
integration jH] with respect to the variable cos 2 9. Motivated by that result Grinberg 
and Rubin |3] define also a cosine of an "angle" for a fc-plane £ and fc'-plane in M n , 
defined as a k x & semi-positive matrix, defined up unitary {7 ( k, K) -equivalence. To 
be more precise we denote M n ^ := M n> fc(K) the space of n x fc-matrices over IK, 
and S nt k = S nt k(y£) the Stiefel manifold of all orthogonal k- frames in K n , identified 
also with the set of all isometric K-linear transformations x G M n ± ■ K fe — > K7\ 
The Grassmannian G n ^ will be viewed as a quotient space of S n k via the mapping 
x G S^fc — > £ = {a;} = xIK fc G G n ,fe, and we will identify a function /(£) on 
as a right U(k, K)-invariant function /(a;) on S n k. We define now, following the 
cosine of the "angle" (77,2?) between the element x G S n> k and 77 G Gr n> fc'(K) to be a 
semi-positive k x fc-matrix, 



where P v is the orthogonal projection K n — > 77. (One can define an angle "(77, as a 
x fc-self adjoint matrix using the spectral calculus of t — > cos 2 1, however we will only 

need Cos 2 (?7, x).) The matrix Cos 2 (?7,£) of the "angle" between 77 G G n ^> and £ G G ni £; 

can be defined up to the U(k,K) equivalence as Cos 2 (t7,x), with £ = {x}. 
The generalization to higher rank spaces of the integration of <j)(rj) is 



where d x r\ is a group-invariant measure on the set of integration. 

Our main result in this paper is Theorem 4.6, it gives an inversion formula expressing 
/(£) in terms the Garding-Gindikin fractional integration I s and the Cayley type 
differential operator acting on the function (Ti </>)(£) with respect to the r- variable. 
This generalizes the result of Grinberg- Rubin for real Grassmannians jHj- The principal 
technical tools in proving it are Propositions IH.HI and IH.5[ giving integral formulas on 
matrix spaces and on Stiefel manifolds. Proposition 13.31 (ii) is proved for the real 
Grassmannian in [3] by using rather complicated formulas involving Bessel functions 
proved by Herz Our idea is instead to find first an integral formula on the matrix 
space, namely Proposition ^. 31 (i). and (ii) will be then a direct consequence. This then 
answers a question in 0. A similar integral formula as our Proposition 3.5 is proved 
in (HI Lemma 2.5] using again some interesting and however tricky computations. Here 
we use again integration on the matrix space. We believe that our proofs are easier, 
both technically and conceptually. Nevertheless it will be clear that we are very much 
inspired by that paper. 

I would like to thank Prof. Boris Rubin for some helpful correspondence and for 
bringing [TJ] to my attention. Thanks are also due to the referees for the careful 
reading and comments on an earlier version of this paper. 



Cos 2 (7/,x) = x*P rj x 




i = to. 



RADON TRANSFORM 



3 



2. Symmetric cones and Garding-Gindikin Fractional Integration 

In this section we fix notations and we recall some known results on the Gindikin 
Gamma function on symmetric cones; see pQ for a systematic treatment. 

Let K = R, C, H be the field of real, complex or quaternionic numbers with real 
dimension 

d := dim M K = 1,2,4 

and let x — > x be the standard conjugation. Let M n> k '■= M n ^(K) be the Euclidean 
space of all n x fc-matrices with entries x in IK. For x G M n ^ we denote x* := x T , where 
T stands for the transpose. The space M n ^ is then a Euclidean space with the metric 
tr(xx*) and the corresponding Lebesgue measure dx. Throughout this paper we will fix 
1 < k < n if nothing else if specified. The vector space K fe will be identified as column 
vectors v with multiplication by c G IK from right, v i— > vc, and all matrices x G M n ^ 
will be identified as IK-linear transformations from IK fc to IK n (i.e. x(vc) = x(v)c, 
v G K h , c G K) unless something else is specified. 
The M-subspace 

A = A k := {a G M Kk \ a* = a} 

of the self-adjoint M^-matrices forms a formally real Jordan algebra with the Jordan 
product aob = \{ab + ba) and the identity I := Ik- Its (real) dimension N is given by 

N ■= N k = dim K ^l = k + ^k(k - 1) 

Let Q = Qk be the cone in A of positive definite k x k K-matrices. Let A be the 
determinant function on A. It is a polynomial of degree k; for real and complex 
matrices it is the usual determinant, for quaternionic matrices it can be defined using 
the Pfaffian by identifying A with a subspace of complex 2k x 2k skew symmetric 
complex matrices. Let GLjt(K) be the group of invertible k x k K-matrices. It acts 
on Q = Qk by g : r — > grg*, and with this action the cone Q becomes a Riemannian 
symmetric space, Q = GLk(K)/U(k, K), where Z7(fc, K) is the orthogonal group over 
IK. The GLfc(K)-invariant measure on Q is 

(2.1) dfc(a) := A(a)- N/k da. 

The Gindikin Gamma integral is defined and given by 

f N 

(2.2) / e- tT(st,) A(s) x dL(s) = T n (X)A(t)- x , t G Q, > — - 1 
Jn k 

where 

r,(A) = (2,r)/ 2 nr(A^(j-i)) 

is the Gindikin Gamma function. (Note that a different normalization of the measure 
ds is used in [3].) The corresponding Beta-integral is 

r s at 

(2.3) / A{t) x - N/k A{s - t)^ N/k dt = B U (X, fi)A{s) x+ ^ N/k , MX, U/i > T - 1 
Jo * 
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with 

n / a v r n (A)r n (//) 

±fo(A,//J = . . 

r Q (A + /i) 

Let (0, 7) be the unit open interval (0, /) = {s 6 fl; s < 7}. The Garding-Gindikin 
fractional integral for a function / on (0, 7) is defined by 

1 f s N 
(I'fM = TTJTT / A(a - t)*- N ' k f(t)dt, 3?A > -, 

Iq(A) Jo k 

where the integration is understood as over the set t 6 O, i < s. For our purpose later 
(see Lemma 4.3) we consider the space 7 1 ((0, 7), A(J - t) u ~Tdt) with a G R. The 
integral (7 A /)(s) is then well-defined for / G L x ((0, 7), A(7 - tf-^dt). 

Using the previous formula it is easy to prove the following semigroup property 

for / G 7^(0, 7), A(7 - t) v -Tdt). 

To define the operator 7 A for smaller 9ftA we need the differential operator A(<9), the 
so-called Cayley type differential operator, defined uniquely on A by requiring that 

A(d x )e tv{xy ' ] = A(y)e tv{xy *\ 

It particular it follows from the equality (|2.2jl that 

k 

A(9)A( s ) A = n(A + ^(j-l))A( s ) A - 1 
i=i 

for any A G C, which sometimes is referred as Cayley-Capelli type identity. 

We state and prove some elementary results on the integral operator 7 A . They may 
have been proved in more general form in the literature. 

Lemma 2.1. (i) Suppose 3?A > y — 1 and \i > y — 1. The operator 7 A defines 
a bounded operator from 7 1 ((0,7), A(7 - ty +5RX -Tdt) into 7 1 ((0,7), A(7 - 

ty-^dt). 

(ii) Suppose z/ > y — 1 and / G 7 1 ((0, 7), A(J — t) u ~~ dt). I x f has an analytic con- 
tinuation in A G C as a distribution on the space Co°(0, 7) of smooth functions 
with compact support in (0,7). 

(iii) Suppose v > f - 1 and / G 7 1 ((0,7), A(7 - tf^^dt). Let m > f be an 
integer. Then 

A(d) m 7"7 = /, 



in the sense of distributions. 
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Proof, (i). Let A and // be as in (i). We estimate the norm of I x f in -^ 1 ((0, I), A (I — 
ty~~£dt). It is, apart from the constant rn(A) _1 , 

/ | / A{s - t) x -% f(t)dt\A(I - sf-Tds 
Jo Jo 

< f [ S A(s-tf x -r \f(t)\A{I- s y-Tdtds 
Jo Jo 

= J* (J*A{s-tf x -%A(I- s y-Tds\ \ f{t)\dt. 

We compute the inner integral. Performing the change of variables s = t + P{{I — 
t)^)i v )> v £ (Oj-0) w here x — > P(x) is the quadratic representation of the Jordan 
algebra A (see 0). We have then ds = A(I - t)%dv, A(I - s) = A(J - t)A(I - v), 
A(s - t) = A(I - t)A(v). That integral is 

A(J - t)"+ RA -£ /" A(v) 5RA -^A(/-t;)^T^ = 5( / u,KA)A(/-t)^ +5RA -T. 

The result follows by substituting this into the previous estimate. 

(ii). To define I x for general A we note that for 9ftA > x- and a test function 
4> G Co°(0, /), we have, viewing the operator A(<9) as acting on distributions 

{A{d)I x fA) = {-l)\l x f,A{d)<P) = {-l?^ j\l x f){s)A{d)<P{s)ds 



:_1)?C r^A) I! ([ A(s " t)X ^ f(t)dt ) A ^^ ds 



/ / 1 ri 



To treat the inner integral (for fixed t) we change variables s — t + u, u £ A and 
denote ip(u) = <p(s) = <fi(t + u). The function ip so defined is then a smooth function 
on A of compact support in the interval (— t, I — t), in particular it is in the Schwartz 
space S(A). Moreover, and A(d s )(p(s) = A{d u )ip{u). The inner integral can be written 

as 

(-^r-TxT f * ^) x -^A{d)^u)du={-l) k -±- j A(u) x -fA(d)^u)du. 

This is the Riesz integral as tempered distribution treated in pQ, VII. It follows from 
Theorem VII. 2.2(h) there that the integral is 

1 

r n (A-i; 

This implies that 

{A{d)l x fA) = {i x ~ l fA)- 

In this way we can define I x f for 9ft A > ^ — 2, and successively for all A. Thus I x has 
analytic continuation for all A G C. 



A{u) x ^~i)(u)du. 
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(iii) By the same computation we have 



Jo V r n (m) J n / 
and it follows again from the first and the third equality in Theorem VII. 2. 2 loc. cit. 
that the inner integral is the delta distribution on ip, 

"n(m) 

proving (iii). □ 

The explicit analytic continuation of I x f is rather complicated; see e.g. [IT] and 
references therein for some systematic study. 

3. Polar decomposition and Bi-Stiefel decomposition 

In this section we prove some integral formulas related to certain polar decomposi- 
tions of matrices. Let S Ujk be the Stiefel manifold of all ortho normal frames in K n . It 
can be realized as the manifold of all isometries x G M n ^. Let 

G := U(n, K) = {g G M n , n , g*g = I n } = 0{n), U(n), Spin) 

be the orthogonal, unitary, and symplectic unitary groups according to K = R, C, H 
respectively. It acts transitively on S njk by the defining action, and thus 

S n ,k = G/U(n - k,K). 

Each x G S n p. defines a fc-dimensional subspace over K, 



N 



£ = { x } := xK k C K n G G 



n, k- 



Thus G Ut k can be identified as the space of orbits in S Ui k under the action of the unitary 
group U(k,K) on K k , 

G n>k = S n>k /U(k,K). 

The manifold G n ^ is therefore a compact Riemannian symmetric space 

G n>k = U(n, K) /U(n - k, K) x U(k, K); 

see e.g. p|. We will later specify certain reference points in S nik and G n ^ thus the 
respective isotropic subgroups. Throughout the paper we will identify functions /(£) 
on with right [/(fc, K)-invariant functions f(x) on the Stiefel manifold S ntk - 

For any compact group K we let o?fc the normalized Haar measure on K. Denote also 
dv and d£ the normalized unique G-invariant measures on S n>k and on G Hjk respectively. 
The following result is possibly known, for completeness we give a proof. 

Lemma 3.1. Let dx and dv, dr be the measures on M n>k , S n>r and Q normalized as 
above. Almost all x G M n>k can be decomposed uniquely as 

x = vr?, v G S n ±, r G Q, 

and under that decomposition the measure dx is given by 

dx = C A(r)^ n dvdt(r) = C A(r^ n - N/k dvdr, 
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namely 

' f(x)dx = C I [ f{vA)A{r)i n dvdL{r) 



where 

-dnk 



7T 



(3.1) C = C (n,k) = 

1 n{2 n ) 

Proof. The polar decomposition follows from the general polar decomposition for linear 
transformations and from the fact that the set of elements of full rank k (over K) is 
an open dense subset of M n ^. Now by the G-invariance we see that there is a weight 
function depending only on r, say W(r), so that 



f(x)dx= / / f(vr 2 )W(r)di(r)dv, 
M n , fc Js„, k Jn 

in term of the invariant measure di on Q. To find the weight function W(r) we note 
that for any g G GLfc(K), the LHS is 



f(x)dx= / f(xg)\det(g) n \dx= / / f(vr*g)\det(g) n \W(r)di(r)dv, 

M n , k JM nik JS n ,k Jtt 

where det(g) is the real Jacobian of the M-linear transformation on K fc = IR dfc , y G 
R dfc = K k yg E K k = M. dk . We perform further the polar decomposition of rag, 

r5 g = u(g*rg)^ , ueU(k, K), 

so that the above integral is 

f(vu(g*rg)^)\det{g) n \W{r)dL{r)dv. 

Performing first the change of variables g*rg — > r on Q and then vu — > u we see that 
it is 

/ / /K^)|det(( ? )"|Vy((( ? *)- 1 r( 7 - 1 )^(r)^, 

by the invariance of and respectively du. Thus the weight factor W(r) transforms 
as 

W(g*rg) = \det(g) n \W(r), 

from which it follows, using the fact that any r G f2 can be written as r = 

# G GLfc(K) (see e.g. pQ), that W(r) = cA(r)2 n for some constant c. The constant can 

be evaluated by taking the function f(x) = e~ tT ^ x * x ^ and using the formula (|2.2|) . □ 

Remark 3.2. Any a; G M n) &, n > k, can be written as x = vr* where r = x*x > 0. 
The element v is generally not unique, and we can always choose v so that v G S n ^- 
We mention also that, polar decompositions, generally speaking, are closely related to 
the Bessel functions, in particular Lemma 3.1 above is much related to the result in 
0, p. 130. 

Proposition 3.3. Suppose 1 < k < k! < n and k + k! < n. 
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(i) Almost all x G M n>k can be uniquely decomposed as 
r in 

x= i s2, ueSk^k, v e S n - k > jk , re (0,/), sett 

_v(l — r) 2_ 

and under that decomposition the measure dx is given by 

dx = C 1 A(r)i fc '- iV / fc A(/ - r)^ n - k ')- N / k A(s)l n ' N ^dudvdrds 

with 



Ci =Ci(n, fc',Jfe) 



fl(|*M(n-fc'))r n (fn) 
(ii) Almost all w G S^fc can be decomposed as 



w 



, ueS k , jk , v G S n ^ k r jk , re (0,1), 



y(I — r) 2 

and under that decomposition the measure dw is given by 
dw = C 2 A(r)i k '- N ' k A(I - r)^- k ')- N / k dudvdr 

with 

For the proof we need another elementary result. 
Lemma 3.4. The mapping 

(r, s) i— > (x, y) = (s 5 rs2, s 5 (/ _ r)s2) 

is a diffeomorphism from (0, /)xO onto VI x f2, and its Jacobian is given by A(s) N / k , 
namely 

dxcfy = A(s) N / k dsdr. 
Proof. We write z = x + y = s^rs^ + s^(I — r)s^ = s. Thus the mapping 

(r, s) i— > (x,z) = (s^rs^,s) 
maps Q x (0, 7) into {(x,z) G f2 x f2; z > x}, and its inverse is given by 

(x,z) — > (r,s) = (z~ixz~2 , x). 
This proves the first part of the statement. For the Jacobian we have that 

di(z)di(x) = dt(s)dL(r) 
by the invariance. Thus, in term of the Lebesgue measure, 
dxdy = dzdx = A(x) N l k A(z) N ' k di{x)di{z) 

= A(s) N/k A(r) N/k dL(x)dL(z) = A(s) N/k drds. 

□ 



We prove now the Proposition. 
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Proof. The proof is done by a change of variables. Conceptually it is clearer to compute 
the integral j M f(x)dx as in the proof of Lemma 3.1, whereas computationally it is 
easier to compute the measure dx, and we will adopt the latter. First of all we write x 
as a 2 x 1-block matrix under the decomposition of K n = K fc ' ©K n_A: ' and then perform 
polar decomposition 



(3.2) 



x 



Xi 

x 2 



1/2 
UlPi 

1/2 
U2P2 



and we may assume that pi > 0,p 2 > and U\ G Sk>,k an d u 2 G S n -k',k, by the 
condition that k' > k, n — k' > k. Thus by Lemma 3.1, 



(3.3) dx = dx\dx 2 , dx\ = cidu\A(pi)^ k * dpi, dx 2 = c 2 du 2 A(p 2 



* dp 2 



with ci = Co(k', k), c 2 = Coin — k', k). On the other hand, applying Lemma 3.1 again 
to x we can write 

x = ws^, w G S n ,k, sGO 

with 



(3.4) 



We write w G S ni k as a block matrix w = 
sition, using the fact that w*w = I = Ik 



dx = c 3 A(s)i n - N/k dwds, c 3 = C {n, k). 

and perform again the polar decompo- 



Wi 

w 2 



1 



(3.5) w 

vyl — r) 

for some r, < r < / and partial isometries u and r> . We can again assume, up to a 
set of dw-measure zero, that < r < /, u G 5fc',fc an d f G S n _k'.k- Therefore x has the 
form 

(3.6) 



1/2 

UlPi 

1/2 
W2P2 



x 2 



x 



1 

WS 2 



1 

UT 2 



u(I — r) 



1 

S 2 



1 1 
r 252 



Thus 
(3.7) 



X\ 



1 1 

wr 2 s 2 . 



x 2 = r(J — r) 2 52 ; p x 
for the change of variables (r, s) 



1 1 
s 2 rs 2 . 



Now using Lemma 
()3.3|) becomes 



p 2 = s 2 (/ — r)s 2 . 
■* (^1,^2), we have that 



dx = dx\dx 2 = cic 2 A(pi) 2 fe * A(p 2 ) 



(n-k')~ 



cic 2 A(pi 
cic 2 A(r 



" fc A(p 2 
kA(I - r 



1 " \ l ) - * du\du 2 dp\dp 2 
^- k ')-fA(s) N / k d Ul du 2 dsdr 



Un-k')- 



A(s)2 n k du\du 2 drds. 



To find the relation between u, v and u%, u 2 we perform further the polar decompositions 
of r2s2 and (/ — r)2g2. They are, in view of (|3.7I ). 



(3. 



1 1 

y2g2 



uop! , (I 



r ) 2 s 2 



VoPi 
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for some u ,v G U(k,K). Thus 

i 11 II i i - 

u\pl — x\ = ur 2 s 2 = uuopl , u\p\ = X2 = v(I — r) 2 s 2 = iWoPj > 

from which it follows that tti = m« , u 2 = ft>o and 

du\du2 = dudv, 

by invariance. We now obtain 

dx = cic 2 A(r)^ fc '~^A(7 - r)^ n - fc ')-f A(s)^ n "f dudvdrds. 

This is our claim in (i), and (ii) follows in turn by using ([3.4)1 . □ 

In the next Proposition we will need another form of polar decomposition that is 
different from the one in Lemma l3~Tl If 1 < k < k' we have that for almost all x G M kjk , 
viewed as linear transformation K fc — > K. k ' — > K. k under the natural embedding and 
projection, that 

x = [0 k ,k>-k s^}u, s eQ u G S k > tk , 

by the usual polar decomposition of x, where Pig stands for the zero p x g-matrix. 
However this factorization is not unique and thus no integral formula is expected as 
that in Lemma f3. II Nevertheless we have the following substitute. First we shall need 
an integral formula, proved in (HI Lemma 2.4] for real Grassmannians, which follows 
easily by the invariance of the measure: For any function / defined on M k>k (M.) 



(3.9) / f(v*u)du= / f(u*v)du= / f(v*u)dv. 

^k' k fe' k As' k 

Proposition 3.5. Let 1 < k < k' . For any measurable function H(w) on M kjk (M), let 
H^s) = A^K*'-^- 1 ))- 1 / H([0 k>k ,_ k s*]v)dv, s G fi. 

and 

H 2 {s) = /±{s)i- 1 f H(us^)du, sen. 

JU{k,K) 

Then for any £ > — 1, 

(3.10) (/ e+iV / fe #i)(s) = C 3 (/^ (fc '" fc)+£+ArA '^2)(s), 
where 

C 3 (n, fc , k = * 

if k' — k > k and 

, / , n C (M' - ^M)^,^) 

Cs(n ' k ' fc) " c^I) 

if k' — k < k. Here Fn fc ,_ fc is the Gamma function associated with the symmetric cone 
flk'-k in Mfc/_fc,fc'_fc and Co(fc, A;' — k) the constant Co in (j3.1J) with (n, fc) replaced by 
(k, k! - k). 
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Proof. We note that it is understood that (|3.1U|) holds whenever one of the integrals is 
absolutely convergent. The idea of the proof is to write the integral P H% as an integral 
on the matrix spaces. Fix r G Q and consider the integral 

~h 




)A(r — w*w) £ dw 



J(r) := / H{w* 

J w€:M k i k :w*w<r 

We write the matrix w* also in block form: w* = \w* w 2 ] with w\ G \ //,-./,. w 2 G 
Mfc/_fc £. We have 



i7(r) = / H{w\) I A(r — u^iui — W2W 2 ) e dw 2 J dwi. 

The inner integral can be computed by 



/ 



A(r - w*^! - w 2 w 2 ) £ dw 2 = c E A(r - w*^!) 64 "^^' fe ) 



lw 2 eM kl _ k j c :w*W2<r~w*w 1 

by the changing of variables w 2 = y(r — w\wi)^ , where 

A(I-y*y) e dy 



which will be evaluated in the end of the proof. The integral J{f) is then, by Lemma 
3.1, 

J{r) = c £ I #K)A(r - w* 1 w 1 )^ k '- k)+£ dw 1 

( 3 - U ) = c e C Q (k, k) f A(r - s)% ^-h)+^ A ^-i ( f H{us^)du] ds 

JO \JueU(k,K) J 

= C £ {I^ k '-^ +e+N ' k H 2 )(r) 



with 



C e = c £ C (k, k)T n (~{k' - k) + e + N/k). 



On the other hand, the integral J(r) can be computed by Lemma 3.1, performing 
the polar decomposition w = vsz, 



J(r) = C (k',k) / A(s)i (fc '- (fc - 1)) - 1 A(r-s) £ 



H(s^ [0 h] v)di 



H(s*v* 



fc'fc 



)dv 1 ds 



and the inner integral, by ()3.9j) . is 



Namely 

J-(r) = C (k', k)T n (e + N/kyP^H^r), 
with H\ given as in the statement. Comparing the two equalities ()3.11|) and (jSJ) we get 

(F+Wh^s) = C 3 (I^ k '- k)+£+N/k H 2 )(s), 
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a 



C (k' } k)T n (e + N/k)- 
Now we evaluate c £ and prove that C3 is as given in the Proposition (and is in fact 
independent of e\). If k' — k > k then by Lemma 3.1 and (|2.H|) . c £ is 

rl j 

C Q (k' - k, k) / A(J - r) £ A(r)^ k '- k) - N/k dr = C (k' - k, k)B n (e + N/k, ~(k' - k)). 
Jo 2 

If k' — k < k, we have again, noticing that for any y G Mk'-k,k, y*y < h is equivalent to 
yy* < Ik'-k, and that A(I k —y*y) = A(I k /_ k — yy*), that the integral can be expressed 
as integration on the interval (0, I k '-k) hi the symmetric cone flk'-k of smaller rank, 

c £ = C (k, k'-k) j A(I k/ - k ~ rYA{r)^ k - l -^ k '~ k -^dr 

J0<r<Iy_ k 

= C (k, k' - k)Bn k ,_ k {e + 1 + \{k' - k - 1), ~k). 

The constant C3 can then be computed by the formula for Cq and by the formula for 
the Gamma function, we leave the elementary and yet intricate computations to the 
interested reader. □ 

4. Radon transform and the inverse transform 

In this section we will prove our main result, finding an inversion formula for the 
Radon transform. 

To simplify notations we will write (with some abuse of notation) U(l) = U (I, K) 
dropping the symbol K. Let 





'Ik 




"0" 


x = 







A 



£ S n ,k, £0 — {^0}) £0 — {^0} £ G n ^ki 

be two reference points in S n ^ and in G n ^k respectively. Similarly, we fix 

G S n ,k', Vo = {yo}, Vo = {xo} e Gn,fc' 









"0" 


yo = 





, m = 


ik' 



two reference points in S n ^' and in G n ^'- Identifying G n ,k with G n> k = G ■ £0 we have 
G n ,k = G/K where 

K = U(n - k) x U(k) = {k = diag(a, 5) G G; (a, 5) G U(n - k) x U(k)} 

is the isotropic subgroup of £ - Correspondingly 

S n ,k = G ■ Xq = G/U(n — k). 

Similarly, G n ^ = G ■ r] = G/K', S n) k> = G ■ y = G/U{n — k'), with 

K' = U(k') x U(n - k') = {k = diag(r, p) G G; (r, p) G U(k') x U(n - k')} 

the isotropic subgroup of 7] . We will henceforth fix these realizations of the groups 
U(k), U{n — k), U(k') and U(n — k') when viewed as subgroups of G = U{n), if no 
ambiguity would arise. 
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For any 77 G G n ^> the subset 

S(rj) = {£ G G n , k ; $ C 77} 

is a totally geodesic submanifold of G ni fc. It is itself a symmetric space with the induced 
metric. For 77 = rjo, 

5(770) = G k ,, k = U(k')/U(k) x U(k! - k) 

where U(k) x U (k' — k) (now with a different realization) consists of the elements in 
G of the form diag(<5, e, I n -k')- F° r any 77 G Gr n ,fc' let 77 = g v r]o with ^ G G, then 

S(rj) =g v S( m ). 

We let d v £ be the unique [/ {k') invariant measure on Sirf) via the above identification. 
We define the Radon transform K : C°°(G nM ) -> C°°{G niW ) by 



(4.1) 

Equivalently it can be defined as 
(4-2) (ft/) (17) = 



S( V ) 



U(k') 



f(9r,r^o)dr. 



Similarly we define the Radon transform on S n k, 



(4.3) 



W)(y) 



f(g y rx )dr. 



U(k') 



It is easy to see that IZf is well-defined and maps C°°{S n ^) to C°°(S nj k')- In particular 
the two formulas agree when acting on / G C°°{G njk ) viewed as right f/(/c)-invariant 
functions on S n>k . 

As explained in the introduction, we shall define the operator 0(77) — > (T r i/2 </>)(£) 
from C°°(G nt k') to C 00 (G nt k), as the integration of functions <fi over planes 77 that are 
of angle (x,rf) so that Cos 2 (x, 77) = uru*, u G U(k), with x G S nj k a representative 
of £; we shall define a slight generalization, T a , for a a G M k> k, a* a < 4 We will 
write the integration as one on the group U(n — k). For that purpose we define the 
following n x n matrices, written in block matrices under the decomposition K™ = 



fc'-fc 



K fe © K 



n—k'—k 



K k , 



J (a) = 



and 



h'-k 















(4 


- a*a) 


1 

2 







a* 










hi—k'—k 










—a 







(4- 


- aa*)2 


a* 










(4 


- a* a) 2 





/„- 


-fc' 


-* 















h'-k 







-(I k - aa* 


)* 










a 



%) 



where a G M fc)fc , a*a < 4 

Some simple observations about those elements are given in the next Lemma. 
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Lemma 4.1. (i) Suppose < a* a < I k . The elements j(a) and h(a) are in 
G = U(n, K), and j(a)" 1 = = j(~a), K a )~ l = K a T = %*)■ 
(ii) The projection Pj( a )rio = j( a )Pvoj( a )* on t° the subspace j(a)r)o = {j(a)yo} G 
G njk > is given by 



h'-k 

I k — a*a (h — a*a)za* 



a(I k — a*a)a aa* 



(hi) For any v G Sfc',fc, we have 



3W 



and 



/i(a) 



^fc'-fc 
(I fc -a*a)i 

On— k'~k,k 

[0 a] v 
[0k,k (4-a*a)3] 

On-fc'-A; 
-ffc'-A; 

a 



(iv) Assume that < aa* < I k . For any v G S k ' tk there exists an I G C/(n — A;) such 
that 



v 

n -fc',fc 



(4.4) j(a) 

where 6 = [0 fcjfc /_ fc a]v G M fejjfc . 



diag(/, I k )h(b)x 



Proof. We prove only the last statement (iv), the remaining are proved by simple 
matrix computations. First we see that h(b)xo is the right hand side of ()4.4|) is, by 
(hi), 



(4.5) 



h{b)xQ 



(4 - a* a) 2 

0n-2fc,fc 

a 



We have the left hand side of (j4.4j) . by the first formula in (iii), is of the form 



V 






0n-fc',fc 




b 



£ S n ,k 



for some q G M n _ k)k , q*q = I k — b*b. By our assumption < a* a < I we see that 
6 = [0 a]v satisfies also b*b < I and which in turn implies that q is of full rank k. 
The polar decomposition of q is then q = u(I k — 6*6)2, with u G S n -k,k, and u can be 
further written as 

. f L. 
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for I G U(n — k). Namely 

j(0 



v 

On— k'k 



diag(Z, I k 



(J fc - 6*6)5 

6 



which is the right hand side (|4.4jl . □ 
We define, for any 6 G Af fc>fc (K), 6*6 < J, the operator T b : C°°(5 n , fc ') -> C°°(S„,fc) 



(4.6) r b 0(x) 



(t>{9xkj{b) 1 y )dk, x = g x x , g x eG. 



'U(n-k)xU(k) 

It maps C co (G ri! k') to C 00 (G ni k) and has the form 



(4.7) 



4>{9ikj{b) l 7] )dk. 

'U{n-k)xU(k) 

The following lemma clarifies the geometric meaning of the above integral. For 
completeness we give an elementary proof. 

Lemma 4.2. Let < r < I = 1^. and r\ G G n ^> and x = g x Xo G S n ^ for g x G G. 
Then Cos 2 (^,x) = SorS^ for some 8q G £/(&) if and only if r\ = g x pj(r^)r] for some 
p = diag(a, 5) G K. 

Proof. Suppose p = diag(a,5) G K and rj = gxpjir^)^ 1 ^. The projection P r] G 
is given by 



and 



P v g x pP.^ rlrj /g x 



Cos 2 (n,x) = x*P n x = x *pP , i n p*xn = 5r5* 



by Lemma 4.1 (ii). 

Conversely, suppose Cos 2 (?7,x) = <5o r <5* := s. We let to = g' 1 ^. We prove that 
lu = pj(r2) _1 r/ for some p & K, which is our claim. Now P v = g x P u g x) and then by 
definition 

S = Cos (?y, x) = X PrqX = XqPjXq. 

The matrix P^ is a self-adjoint matrix on IK* 1 , and under the decomposition of K n = 
K n ~ k © K fc it is of the block-matrix form 

A B 

B* s 

by the previous formula, with A self-adjoint and B G Mn-^^K). Being a projection, 
P 2 = P u , that is 

A 2 + BB* = A, AB + Bs = B, B*B + s 2 = s. 

The polar decomposition of B is of the form, under the decomposition W n ~ k = K. k '~ k © 
K k ®K n - k ~ k ', 



B = atx sa(4-s)2 




«i G f/(n - fc), 
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for B*B = s(I k - s). 

Let A\ = a 1 1 Aax- Writing A\ in block matrix form under the same decomposition 
and using the equation A 2 + BB* = A we see that 



A n A 13 

(s)3(I fc -s)3 
A 33 



4* 



and that the matrix 



Axx A 13 

A*X3 ^33. 

is a projection of rank k' — k since is of rank k! . Thus 







= «2 


Ik'-k 


0" 


/M3 


^33_ 









for some a<i G £/(n — 2A;). Finally we have 

^ = pjMr 1 ^ ) oj(^)p* 

with p = diag(ai«2, S) G if, which is equivalent to 

as any subspace to is uniquely determined by the projection P u 



□ 



For < r < I the operator Ti can be expressed in terms of the angle explained in 



the introduction, 



<P(y)d x y. 

'r;={j/}: Cos 2 (r],x)=uru* ,u£U (k) 

For the real Grassmannians this is the slightly corrected version of the formulas (1-8) 
and (3.2) in [3] (uru* should be in place of r there). 

To state the next proposition we need also a mean- value operator, Wb '■ C°°(S njk ) —> 
C°°(S ntk ), for beM ktk , b*b<I, 



(4 



f(g x diag(a, I k )h(b)x )da; 

U(n-k) 

again W a is well-defined. Observe that it does not map functions on G n ,k to functions 
G n ,k- If / e C°°(G njk ), the function 

(W Sa f)(x)d5 

U(k) 

is right U(k) -invariant and thus defines a functions on G njk . (For k = 1 the operator 
W^a can roughly speaking be incorporated into the operator T so that it doesn't play 
a role.) 

First we state an integral formula, which is a direct consequence of Proposition 3.3 
(ii), written in the form 



w 



v(I — r) 2 



i 

ur 2 



veS n „ k>k , ueU(k), re (0,1), 
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and of the formula ([4.5)1 : see also PI Lemma 3.5] (a different formula for the mean- value 
operator is defined there). 



Lemma 4.3. Suppose 1 < p < oo and / G L p (S n ^). Then for any x G S Ut 



f(g x w)dw = / f(w)dw 
C 2 {n,k) f k /\{h-r)i {n - 2k+l ^ l A{r)i- 1 [ I (W if){x)du)dr. 



U(k) ur ^' 



In particular, the function 



(4.9) $o(r) = $ (r,x):=A(r)2- 1 / (W if)(x)du 

Ju(k) ur 

is in ^((0, 4), A(/ & - r )iin-2k + i)-i dr y 

Proposition 4.4. Let e G C. Suppose / G L 1 (S , „ i jt) and = 7?./. Then, in the sense 
of distribution, 

P+N/k^ = c 3 jf(fc / -*)+=+JV/*$ 0j 
as analytic continuation in £, where 

$ 1 ( S ) = $ 1 (^)=A(^( fc '-(MH (T 

and $o is as i n Q4.9J1 . 

Proof. Let r G (0,/). We compute (Ti0)(x) = (T iTZf)(x). We prove the formula 
for e > 0, in which case the convergence of the relevant integrals are easily justified by 
using Lemma 14. 3| and the result then follows for any e by analytic continuation. 
By the definitions of T and TZ we have, 

CO 0) 0*0 = f{g x pj{r^)- 1 TXQ)dTdp, 

JKJT&U(k') 

where f = diag(r, I n -k) in the formula. Introducing the variable v = rlk G Sk>,k, we 
have 

~ _ v 
TX ° ~ ' 

furthermore using Lemma 4.1 (iv) we have 

j(r 1/2 )~ 1; rx = diag(Z,4)/i([0 r^]v)x , 
for some I G U(n — k). Now p = diag(a, 5), 

pj{r l l 2 )~ l TXQ = diag(a,5) diag(7, lk)h([0 r^]v)xo = diag(oi, S)h([0 r^]v)xo, 
thus changing variables al — > a the previous integral becomes 

/ / / f(g x di&g(a,I k ) diag(/ n _ fe , 6)h([0 r*]v)x ))d8dadv 

JSy )k Ja€U(n-k)) J SeU(k) 

/ / f(9x diag(a, Ik)h(5[0 r ^]v)x ))dSdadv . 

Sy u Ja£U(n-k)) J5£U{k) 
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In term of the mean-value operator the integral can be written as 
(4-10) (r pJ *)(s)=/ / (W h f){x)dSdv 

We use now Proposition 3.5 with H{w) = f SEU r k }(Wswf){ x )i obtaining 
(r+ N / k ^)(s) = C 3 (ll (k '~ k)+£+N/k <S>o)(s), s G (0, I). 



□ 



The next lemma is elementary, the first two parts can be proved by the same method 
as that of Lemma 3.4 in [3]. The last part is proved by using the triangle inequality 
and the fact that A(s) < 1 for s G (0,7); we omit the proof. 

Lemma 4.5. (i) For any / G L p (S n ^), 1 < p < oo, we have 

lim W a f = f- 

a — >I k ,a* a<I k 

where the limit is in L p (S , rii fc)-sense. 
(ii) If / G W(G n , k ) then 

lim / (W Sa f)d5 = f. 



a— >Ik,a*a<Ik 



U(k) 



and the limit is taken in L p (G n> k)- 
[iii) If / G L p (S nt k) or L p (G nt k) then for any complex number (, 



lim A(a*a)( [ {W Sa f)d5 = f. 

k ,a*a<I k JU(k) 



a — >I k ,a*a<I k 

and the limit is taken in the respective spaces. 

Theorem 4.6. Let K = R, C, H be the field of real, complex and quaternionic numbers. 
Let l<k<k'<n — 1, k + k' < n. Let A (d) be the Cayley-type differential operator 
acting on the space of self-adjoint k x fc-matrices over K and I x be the Gar ding- Gindikin 
fractional integration. Suppose 1 < p < oo, / G L p (G nj fc(K)) and = TZf. Let m be 
such that 

m > ~(k' - 1). 

Then the Radon transform / — > <ft = TZf is inverted by the formula 

A(tf s }'".T"^" ' 

where the limit is taken in the space L p (G nj fc(IK)) and the differential operator acts in 
the sense of distribution, and where 

s2 ' 



/ = lim A{d s ) m I m -^ k '- k) $ 



Proof. Let m be as in the statement and let £ > —1 be given so that m = |(A/ — k) + 
e + iVy/c. Then Proposition 4.4 can be restated as 



(/™$ )( s ) = (J m -f( fc '- fc $)( s 
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Now Lemma 4.3 implies that $o is i n a -^-space, and we can then use Lemma 2.1 (hi), 
obtaining 

A(s)!- 1 [ (W if)(x)dd = $ (s) = A(d s ) m (I m ~^ k '- k ^)(s), 
JU(k) us 

in the distributional sense. The result then follows from Lemma [4.51 (iii). □ 

There arise several interesting questions such as inverting Radon transform on non- 
compact symmetric matrix domains and finding a Plancherel formula. In a forthcoming 
paper we will generalize the results in this paper to that setup. 
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